16409 do not use receive step to determine root report #16723

jack-h-wang · 2024-12-05T20:38:36Z

This PR adjusts the search for root reports to not rely on explicitly searching for the receive step report. Additionally, the root report from the submissions endpoint should now include the sender-supplied payloadname if present.

Test Steps:

The tests in ReportGraphTest and FhirTranslatorIntegrationTests are sufficient

Changes

Do not use receive step as indicator of a root report.
- Instead, a root report is determined by a report ID that does not appear in the report_lineage table as a child report.
Retrieve payloadname from submission endpoint message headers when creating root report.

Checklist

Testing

Tested locally?
Ran ./prime test or ./gradlew testSmoke against local Docker ReportStream container?
(For Changes to /frontend-react/...) Ran npm run lint:write?
Added tests?

Linked Issues

Fixes Identify and update DB queries to be independent of the Receive step #16409

github-actions · 2024-12-05T20:38:50Z

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details

Scanned Manifest Files

github-actions · 2024-12-05T20:50:35Z

Test Results

1 258 tests +2 1 254 ✅ +2 7m 27s ⏱️ -10s
164 suites ±0 4 💤 ±0
164 files ±0 0 ❌ ±0

Results for commit 5e8a3c3. ± Comparison against base commit 986d6fb.

This pull request removes 2 and adds 4 tests. Note that renamed tests count towards both.

gov.cdc.prime.router.fhirengine.azure.FHIRTranslatorIntegrationTests ‑ successfully translate HL7 for FHIR receiver when isSendOriginal is true()
gov.cdc.prime.router.fhirengine.azure.FHIRTranslatorIntegrationTests ‑ successfully translate for FHIR receiver when isSendOriginal is true()

gov.cdc.prime.router.fhirengine.azure.FHIRTranslatorIntegrationTests ‑ successfully translate HL7 for FHIR receiver when isSendOriginal is true from convert step()
gov.cdc.prime.router.fhirengine.azure.FHIRTranslatorIntegrationTests ‑ successfully translate HL7 for FHIR receiver when isSendOriginal is true from receive step()
gov.cdc.prime.router.fhirengine.azure.FHIRTranslatorIntegrationTests ‑ successfully translate for FHIR receiver when isSendOriginal is true from convert step()
gov.cdc.prime.router.fhirengine.azure.FHIRTranslatorIntegrationTests ‑ successfully translate for FHIR receiver when isSendOriginal is true from receive step()

♻️ This comment has been updated with latest results.

github-actions · 2024-12-05T21:15:54Z

Integration Test Results

60 files ±0 60 suites ±0 36m 59s ⏱️ -15s
424 tests ±0 414 ✅ ±0 10 💤 ±0 0 ❌ ±0
427 runs ±0 417 ✅ ±0 10 💤 ±0 0 ❌ ±0

Results for commit 5e8a3c3. ± Comparison against base commit 986d6fb.

♻️ This comment has been updated with latest results.

jack-h-wang · 2024-12-11T17:13:35Z

prime-router/src/main/kotlin/fhirengine/engine/FHIRConverter.kt

@@ -178,7 +180,8 @@ class FHIRConverter(
                // is properly recorded in the report file table with the correct sender
                actionHistory.trackExternalInputReport(
                    report,
-                    BlobAccess.BlobInfo(format, blobUrl, blobDigest.toByteArray())
+                    BlobAccess.BlobInfo(format, blobUrl, blobDigest.toByteArray()),
+                    payloadName


This change to carry over payloadname from the submissions API is needed to continue supporting the sendOriginal function. This adds the information to the root report being created here.

jack-h-wang · 2024-12-11T17:20:53Z

prime-router/src/main/kotlin/history/db/ReportGraph.kt

+            .leftJoin(REPORT_LINEAGE)
+            .on(REPORT_FILE.REPORT_ID.eq(REPORT_LINEAGE.CHILD_REPORT_ID))
+            .where(REPORT_LINEAGE.PARENT_REPORT_ID.isNull())
+            .orderBy(REPORT_FILE.ACTION_ID.asc())


The previous query was to join the action table in order to check if the action_name is receive. The new query instead filters the list of reports to only include those that do not appear in report_lineage as a child. This list is then ordered by action_id to ensure the results are output in a consistent order. The join to the action table is removed since it is no longer necessary.

Thanks, this explanation was very helpful!

Just to clarify, we are now recursively going up the report lineage table until we find a parent report id which is null?

The recursive query (cte) was not altered, but that is the current and previous behavior. The change here was in the way the results from the recursive query are filtered.

Got it 👍

jack-h-wang · 2024-12-11T17:24:43Z

prime-router/src/test/kotlin/fhirengine/azure/FHIRTranslatorIntegrationTests.kt

@@ -798,4 +798,166 @@ class FHIRTranslatorIntegrationTests : Logging {
            assertThat(translatedValue).isEqualTo(reportContents.toByteArray())
        }
    }
+
+    @Test
+    fun `successfully translate HL7 for FHIR receiver when isSendOriginal is true from convert step`() {


These tests ensure sendOriginal items still function when convert is the root report, as the case would be when the submissions API is used.

JFisk42

Code looks good to the best of my understanding.
I ran the tests locally and everything is as expected.

Side note: I wanted to use this PR as an excuse to try out the new submission microservice. I was not able to get any reports sent to it because of setup issues. Most likely this is because of the extra steps currently required to get the authorization service running locally. This should no block this PR.

jalbinson · 2024-12-13T18:29:56Z

prime-router/src/main/kotlin/fhirengine/engine/FHIRConverter.kt

@@ -111,6 +111,7 @@ class FHIRConverter(
        companion object {

            private val clientIdHeader = "client_id"
+            private val payloadNameHeader = "payloadname"


This doesn't need to be done now, but since in the future these header names will be used across multiple services they should be defined in the shared project. It makes it less prone to errors if we have to change them for some reason in the future.

Maybe you could throw a TODO on there with your ticket for submissions service updates.

…appear as child in report_lineage

sonarqubecloud · 2024-12-13T20:19:20Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

This reverts commit 6cb62ec.

…" (#16834) This reverts commit 6cb62ec.

jack-h-wang temporarily deployed to staging December 5, 2024 20:38 — with GitHub Actions Inactive

jack-h-wang added the platform Platform Team label Dec 6, 2024

jack-h-wang temporarily deployed to staging December 6, 2024 20:40 — with GitHub Actions Inactive

jack-h-wang temporarily deployed to staging December 9, 2024 16:55 — with GitHub Actions Inactive

jack-h-wang force-pushed the platform/jwang/16409-query-root-report-without-receive branch from 5c35630 to 4f56b4b Compare December 9, 2024 17:21

jack-h-wang temporarily deployed to staging December 9, 2024 17:22 — with GitHub Actions Inactive

jack-h-wang temporarily deployed to staging December 9, 2024 20:03 — with GitHub Actions Inactive

jack-h-wang temporarily deployed to staging December 11, 2024 03:31 — with GitHub Actions Inactive

jack-h-wang marked this pull request as ready for review December 11, 2024 03:37

jack-h-wang requested a review from a team as a code owner December 11, 2024 03:37

arnejduranovic requested review from adegolier and JFisk42 December 11, 2024 17:11

jack-h-wang commented Dec 11, 2024

View reviewed changes

jack-h-wang temporarily deployed to staging December 11, 2024 17:27 — with GitHub Actions Inactive

jack-h-wang mentioned this pull request Dec 11, 2024

Submissions endpoint stores additional metadata #16727

Closed

3 tasks

jack-h-wang force-pushed the platform/jwang/16409-query-root-report-without-receive branch from d665d9b to b7c1982 Compare December 11, 2024 23:52

jack-h-wang temporarily deployed to staging December 11, 2024 23:52 — with GitHub Actions Inactive

arnejduranovic assigned JFisk42 Dec 12, 2024

JFisk42 approved these changes Dec 13, 2024

View reviewed changes

jalbinson reviewed Dec 13, 2024

View reviewed changes

jalbinson approved these changes Dec 13, 2024

View reviewed changes

jack-h-wang added 4 commits December 13, 2024 14:56

replace check for receive step with check for report_ids that do not …

835675c

…appear as child in report_lineage

remove unnecessary join to action table

540f505

track payloadname for submissions from the submissions api

02079e6

add tests for sendoriginal from convert step

95839ba

jack-h-wang added 2 commits December 13, 2024 14:56

order by action_id for consistent result ordering

2074833

rename variable for clarity

5e8a3c3

jack-h-wang force-pushed the platform/jwang/16409-query-root-report-without-receive branch from b7c1982 to 5e8a3c3 Compare December 13, 2024 19:56

jack-h-wang temporarily deployed to staging December 13, 2024 19:57 — with GitHub Actions Inactive

jack-h-wang merged commit 6cb62ec into main Dec 13, 2024
25 checks passed

jack-h-wang deleted the platform/jwang/16409-query-root-report-without-receive branch December 13, 2024 21:20

jack-h-wang added a commit that referenced this pull request Dec 18, 2024

Revert "16409 do not use receive step to determine root report (#16723)"

556947c

This reverts commit 6cb62ec.

jack-h-wang mentioned this pull request Dec 18, 2024

Revert "16409 do not use receive step to determine root report" #16834

Merged

jack-h-wang added a commit that referenced this pull request Dec 18, 2024

Revert "16409 do not use receive step to determine root report (#16723)"

cf94225

This reverts commit 6cb62ec.

jack-h-wang added a commit that referenced this pull request Dec 18, 2024

Revert "16409 do not use receive step to determine root report (#16723)…

b6774b8

…" (#16834) This reverts commit 6cb62ec.

jack-h-wang mentioned this pull request Dec 18, 2024

Identify and update DB queries to be independent of the Receive step #16409

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

16409 do not use receive step to determine root report #16723

16409 do not use receive step to determine root report #16723

jack-h-wang commented Dec 5, 2024 •

edited

Loading

github-actions bot commented Dec 5, 2024 •

edited

Loading

github-actions bot commented Dec 5, 2024 •

edited

Loading

github-actions bot commented Dec 5, 2024 •

edited

Loading

jack-h-wang Dec 11, 2024

jack-h-wang Dec 11, 2024

JFisk42 Dec 13, 2024

jalbinson Dec 13, 2024

jack-h-wang Dec 13, 2024 •

edited

Loading

jalbinson Dec 13, 2024

jack-h-wang Dec 11, 2024

JFisk42 left a comment

jalbinson Dec 13, 2024

sonarqubecloud bot commented Dec 13, 2024

16409 do not use receive step to determine root report #16723

16409 do not use receive step to determine root report #16723

Conversation

jack-h-wang commented Dec 5, 2024 • edited Loading

Changes

Checklist

Testing

Linked Issues

github-actions bot commented Dec 5, 2024 • edited Loading

Dependency Review

OpenSSF Scorecard

Scanned Manifest Files

github-actions bot commented Dec 5, 2024 • edited Loading

Test Results

github-actions bot commented Dec 5, 2024 • edited Loading

Integration Test Results

jack-h-wang Dec 11, 2024

Choose a reason for hiding this comment

jack-h-wang Dec 11, 2024

Choose a reason for hiding this comment

JFisk42 Dec 13, 2024

Choose a reason for hiding this comment

jalbinson Dec 13, 2024

Choose a reason for hiding this comment

jack-h-wang Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

jalbinson Dec 13, 2024

Choose a reason for hiding this comment

jack-h-wang Dec 11, 2024

Choose a reason for hiding this comment

JFisk42 left a comment

Choose a reason for hiding this comment

jalbinson Dec 13, 2024

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 13, 2024

Quality Gate passed

jack-h-wang commented Dec 5, 2024 •

edited

Loading

github-actions bot commented Dec 5, 2024 •

edited

Loading

github-actions bot commented Dec 5, 2024 •

edited

Loading

github-actions bot commented Dec 5, 2024 •

edited

Loading

jack-h-wang Dec 13, 2024 •

edited

Loading